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INTRODUCTION 

Twenty-two  years  ago,  at  a  similar  podium  down  the  street  in  Gulf  Shores 
Alabama,  we  reviewed  the  impact  of  then-recent  technological  advances  in 
collecting  occupational  survey  data.  At  that  1976  Annual  Conference  of  the  Military 
Testing  Association  (MTA),  we  reported  on  lessons  learned  in  migrating  from 
low-tech  "fill  in  the  blank"  occupational  survey  forms  requiring  manual  data  entry 
(keypunch)  to  high-tech  "fill  in  the  bubble"  (mark  sense)  optical  scan  booklets 
(Weissmuller  &,  Kaufman,  1976).  We  found  the  two  standard  types  of  problems  - 
logistical  and  substantive.  The  logistical  type  of  problem  is  associated  with 
capturing  the  respondent’s  input  and  facilitating  its  faithful  transfer  across  space 
and  time  into  a  master  database.  These  problems  are  typically  overcome  by 
adopting  revised  respondent  instructions,  implementing  new  guidelines  for 
instrument  design,  improving  quality  control  procedures  and  creating  custom 
software  to  avoid  error-prone  human  processes. 

The  substantive  type  of  problem  is  associated  with  uncertainties  from  switching 
elicitation  and  recording  methodologies.  In  other  words,  "Does  the  bubble-marking 
process  systematically  affect  the  quality  of  the  data?"  In  these  substantive  issues 
one  is  obliged  to  conduct  a  careful  scientific  analysis  to  ensure  continued  (or 
improved)  validity.  The  transition  impact  on  the  1970s  survey  respondent  was 
minimal  and,  after  minor  logistical  adjustments,  comparable  data  from  old  to  new 
systems  were  easy  to  demonstrate.  The  optically  scanned  occupational  survey 
booklet  became  the  standard  for  the  high-volume  Job  Incumbent  (JI)  data,  with 
keypunched  booklets  being  reserved  for  low-volume  supplementary  ratings  from 
Subject  Matter  Experts  (SMEs)  such  as  Task  Learning  Difficulty  (TD)  and 
Training  Emphasis  (TE). 


The  transition  reviewed  today  is  a  much  more  complicated  situation.  The  present 
goal  is  more  than  moving  occupational  survey  technology  from  optically  scanned 
survey  booklets  of  the  1970s  to  computer-diskette  surveys  of  the  mid-1990s 
(Mitchell,  J.  L,  Weissmuller,  J.  J.,  Bennett,  W.  R.,  Agee,  R.  C,  &  Albert,  W.  C., 
1995).  Today’s  complex  environment  is  mandating  a  suite  of  survey  technologies 
using  ever-evolving  multi-method  approaches  including  combinations  of 
paper-and-pencil  (manual  data  entry),  optical  scan  sheet,  computer-diskette,  email, 
local  area  network,  and  Internet  surveys  (Stanton,  1998).  The  goal  is  to  bring  all 
available  tools  and  techniques  to  bear  on  the  three  pillars  of  occupational  survey 
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validity:  maximum  population  coverage,  effective  elicitation  of  information,  and 
faithful  linkage  of  data  into  the  master  database. 

WHY  MULTI-METHOD  SURVEYS? 


Unique  Jobs  are  Everywhere 

In  designing  new  systems,  one  must  be  mindful  of  the  special  requirements  of  the 
job  at  hand.  The  focus  here  is  on  servicing  occupational  analysis  (OA)  surveys  that, 
as  a  group,  have  very  special  needs.  Occupational  analysis  surveys  target  individual 
job  incumbents  within  a  job  family  because  every  position  is  potentially  unique. 
Standard  personnel  tracking  systems  tend  to  track  individuals  by  "job  title"  and 
"paygrade"  in  a  job  family.  In  many  cases,  job  titles  are  not  highly  reliable 
indicators  of  actual  job  content.  Moreover,  as  jobs  change  with  new  tasks  or  new 
technologies  over  time,  the  job  titles  are  not  always  adjusted  to  reflect  changed 
content.  When  surveys  are  constructed  for  only  a  single  purpose,  say  common 
training  for  an  entire  job  family,  or  training  evaluation  for  a  specific  course,  the 
standard  personnel  tracking  system  may  be  sufficient  to  support  rational,  stratified 
sampling  (Tsai,  1993;  Mitchell,  Tucker,  Fast,  Bennett  &  Albert,  1997).  Such 
focused,  single-purpose  surveys  often  do  not  reveal  changing  job  content,  nor  are 
they  designed  to  do  so.  Further,  as  training  budgets  shrink  and  we  need  to  target 
training  so  as  to  optimize  preparation  for  specific  jobs  (subgroups,  etc.),  then 
generalized  training  surveys  may  not  be  sufficient. 

One  goal  of  responsible  occupational  analysis  is  to  undercover  reliable,  identifiable, 
subgroups  with  stable  commonalties  that  can  be  used  for  recruitment,  selection, 
training,  training  evaluation,  promotion,  restructuring,  multi-level  organizational 
modeling,  up  sizing,  and  out  placement.  When  occupational  data  are  used  for 
applications  across  the  entire  personnel  spectrum,  discussions  about  appropriate 
sampling  levels  become  less  than  productive.  The  basis  of  scientific  sampling  is 
knowledge  of  the  population’s  distribution  across  underlying  dimensions  -  and  the 
purpose  of  occupational  analysis  is  to  help  establish  the  identity  of  those 
dimensions.  For  this  same  reason,  the  survey  methods  must  be  capable  of  surveying 
personnel  in  all  locations,  not  just  central-site  office  workers  who  tend  to  have 
computers  and  hence,  paper  and  pencil  forms  may  never  go  out  of  style. 


OA  Surveys  are  Long  &  Complex 

OA  surveys  tend  to  be  very  large,  typically  1,000  to  2,000  responses.  OA  surveys  are 
growing  in  size  and  are  servicing  increasingly  diverse  job  families.  The  computer’s 
ability  to  support  complex  branching  within  a  survey  is  a  new  opportunity  that  will 
save  countless  hours  at  the  computer  screen  for  future  survey  respondents  (Phalen, 
Tucker,  &  Weissmuller,  1993).  Paper  and  pencil  surveys  can  mimic  this  ability  by 
embedding  written  routing  instructions  like  programmed  instruction  texts  of  old. 
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Whereas  routing  instructions  in  paper  and  pencil  forms  can  be  misapplied  or 
ignored  by  survey  respondents,  in  the  computerized  forms  compliance  is  automated 
and  immediate.  Perhaps  more  importantly,  reproduction  costs  will  limit  the 
enthusiasm  of  the  agency  conducting  the  survey  and  the  physical  size  of  a 
"programmed  instruction"  survey  in  the  person’s  in-basket  will  crush  the 
enthusiasm  of  the  survey  recipients. 

CONCESSIONS  TO  REALITY 

In  the  past,  a  multi-method  survey  usually  meant  that  job  incumbents  were 
surveyed  with  optical  scan  sheets  and  SMEs  were  surveyed  with  manually  entered 
paper  and  pencil  booklets.  In  today’s  OA  environment,  a  multi-method  survey 
would  mean  that  a  single  class  of  respondents  (e.g.,  job  incumbents)  could  receive 
the  total  spectrum  of  survey  instruments  -  based  on  their  expected  level  of  access  to 
computers  and  the  Internet.  In  a  perfect  world,  all  job  incumbents  or  SMEs  could 
use  the  Internet  and  respond  (at  almost  no  reproduction/shipping  cost)  to  a  new 
survey  overnight.  In  reality,  however,  Internet,  Email  &  Local  Area  Network 
(LAN)  access  is  not  yet  universal.  Not  every  individual  has  a  computer  on  his  or  her 
desk  and  scheduling  access  to  a  computer  with  a  3.5"  disk  drive  may  take  some 
time.  In  some  cases  where  job  incumbents  are  deployed  away  from  the  home 
station,  paper  and  pencil  may  present  the  best  approach  to  collecting  data  in  the 
trenches. 

Unfortunately,  the  most  portable  survey  format  (the  manually  entered  paper  and 
pencil  format)  is  the  most  costly  to  data  enter  and  delays  the  analysis  phase  of  the 
project.  The  optical  scan  sheet  approach  is  relatively  fast  in  creating  a  data  base 
from  the  returned  booklets,  but  the  booklets  themselves  are  expensive  and  involve  a 
greater  lead-time  for  set  up.  Local  area  networks  and  email  systems  tend  to  present 
a  host  of  site-specific  barriers  to  automated  survey  distribution  and  data  flow-back. 
Although  telephone  survey  systems  may  be  practical  for  "quick  response"  attitude 
surveys,  the  typical  operational  Occupational  Analysis  (OA)  survey  contains  well  in 
excess  of  1,000  items  and  thereby  precludes  use  of  the  telephone  in  this  area. 

THE  VALIDITY  CONUNDRUM 

In  the  past  we  were  content  to  demonstrate  that  Method  "A"  produced  results 
comparable  to  Method  "B"  and  Method  "B"  was  declared  to  be  validated.  In 
today’s  multi-method  approach,  however,  we  are  not  necessarily  replacing  Method 
"A"  in  favor  of  Methods  "B"  or  "C",  or  "D",  we  are  simply  adding  those  new 
methods  into  our  survey  toolbox.  The  intent  is  to  use  differing  methods 
concurrently  to  achieve  maximum  coverage,  not  simply  use  these  tools  separately  in 
discrete  projects.  It  has  been  shown  that  given  parallel  forms  job  incumbents  tend 
to  check  off  6%  to  8%  more  tasks  on  computerized  occupational  surveys  than  in 
comparable  paper  and  pencil  formats.  Does  this  matter,  given  that  overall  group 
job  descriptions  still  match  with  a  correlation  of  0.98?  ^udspeth  &  Fayfitch,  1990) 

When  some  survey  methods  have  new  opportunities  that  other  methods  can’t 
duplicate,  thought  needs  to  be  given  to  merging  multi-method  data  into  a  master 
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database.  In  a  recent  example  where  new  opportunities  are  exploited,  a  computer 
administration  system  was  used  to  provide  raters  with  real-time  feedback  on  actual 
time-spent  ratings.  A  paper-and-pencil  "parallel"  form  would  not  be  able  to  limit 
or  interactively  guide  a  rater’s  responses  and  without  this  "rater  training"  gross 
errors  in  time  ratings  are  the  expected  norm  (Phalen,  1995).  The  dilemma  then  is, 
should  the  coverage  pillar  be  sacrificed  and  ignore  job  incumbents  who,  because  of 
their  assignment  location,  could  only  use  paper  and  pencil  forms,  or  should  some 
analytical  accommodation  be  developed  to  recognize  and  integrate  all  rating 
paradigms  into  the  master  database? 


Self-Prioritized  Inventory 

A  disk-based  survey  of  20,000  cases  was  conducted  in  April.  It  was  deemed 
important  to  both  maximize  response  validity  (by  combating  fatigue  effects)  and 
minimize  job  incumbent  time  spent  taking  the  900  item  survey.  A  two-level 
branching  design  was  used.  The  master  survey  was  pre-coded  for  items  appropriate 
to  each  of  five  paygrade  groups.  Once  a  person  indicated  his  or  her  paygrade,  the 
appropriate  duty  areas  were  displayed  and  rated  on  an  eight  point  "importance  to 
job"  scale.  After  the  duty  ratings,  the  survey  software  presented  tasks  under  each 
selected  duty,  sequencing  from  highest  rated  duty  to  lowest  rated  duty  area.  This 
effort  was  successful  in  reducing  the  number  of  items  each  individual  had  to  rate, 
and  in  controlling  the  total  amount  of  time  required  to  complete  the  survey. 

In  a  recent  study  of  Air  Force  Security  Force  patrolmen,  incumbents  were 
presented  each  major  duty  area  in  ordered  sequence  based  on  their  own  ratings  of 
the  degree  to  which  the  duties  were  "part  of  the  job"  (defined  as  time  and  effort 
required).  This  ensured  that  the  main  focus  of  the  work  was  rated  first  (Holt, 
Mitchell,  &  Zuniga,  1998). 

Early  work  on  exploiting  new  scales  within  the  automated  survey  environment  also 
validated  the  technique  of  having  job  incumbent  rate  the  importance  of  duty  areas 
then  present  tasks  under  that  self-prioritized  order  (Dittmar,  M,  Hand,  D.,  Tucker, 
D.,  &  Weissmuller,  J.  1995).  In  that  study  it  was  shown  that  there  is  a  statistically 
significant  correlation  between  the  "importance"  ratings  and  the  actual  number  of 
tasks  selected  within  a  duty.  A  second  phase  of  the  present  study  is  collecting 
Training  Emphasis  ratings  from  raters  in  both  automated  and  paper  and  pencil 
versions.  As  raters  are  expected  to  rate  all  tasks,  no  branching  is  used  in  either  the 
automated  or  paper  and  pencil  forms  and  parallel  results  are  expected. 


Lessons  Learned 

In  a  special  purpose,  90-item  job  satisfaction  survey,  200  job  incumbents  were 
surveyed  in  each  of  two  job  families  (Weissmuller,  Grimes,  Siem  &  Kenny,  1997). 
Every  person  surveyed  was  sent  a  diskette  with  the  computer-based  survey  and  a 
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printed  copy  of  the  survey.  Incumbents  were  given  the  option  of  completing  and 
returning  either  form.  In  both  cases,  51%  of  the  returns  were  on  diskette.  The  other 
49%  had  to  be  data  entered.  The  data  entry  was  accomplished  by  using  a  survey 
diskette  and  "transcribing”  the  answers  per  the  prompts  on  the  screen.  The  lessons 
learned  from  this  include  the  fact  that  the  people  who  used  the  paper  and  pencil 
forms  ignored  limits  and  instructions  on  the  form.  The  automated  survey  format 
used  for  "data  entry"  had  to  be  modified  to  record  "invalid"  responses  coming  in 
from  the  paper  versions. 

An  observation  was  made  that  automated  surveys  had  longer  "write-in"  comments 
than  paper  and  pencil  surveys.  No  demographic  differences  were  found  between  the 
groups  selecting  "disk"  or  "paper".  Various  reasons  have  been  offered  to  explain 
the  difference  with  the  most  compelling  being  that  the  incumbents  viewed  the 
survey  disk  as  more  "secure"  for  personal  comments.  If  this  is  the  case,  then  it  is 
the  true  that  incumbents  view  the  media  as  qualitatively  different  -  and  this  speaks 
to  the  relative  validity  of  data  collected. 

End-Game  Requirement 

It  is  essential  that  occupational  survey  methods  are  used  to  maximize  coverage, 
elicit  the  highest  quality  information,  and  accurately  transfer  all  into  a  unified 
database.  The  design  challenge  here  is  not  to  design  a  single  survey  methodology  to 
meet  these  requirements,  but  to  design  an  integrative  methodology  which  can 
simultaneously  draw  on  one  or  more  of  these  tools  in  a  single  survey  to  feed 
comprehensive  and  valid  data  to  a  master  database.  The  analytical  challenge  is  to 
accept  inputs  from  various  sources  and  adjust  the  ultimate  data  received  to  reflect 
the  "perspective"  from  which  it  was  gathered. 
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